Diagnostic performance between in-house and commercial SARS-CoV-2 serological immunoassays including binding-specific antibody and surrogate virus neutralization test (sVNT)

This study aimed to evaluate the correlation between in-house and commercial binding-specific IgG antibodies and between in-house and commercial SARS-CoV-2 surrogate virus neutralization tests (sVNT). Samples from healthcare workers who received vaccines against SARS-CoV-2 were tested for RBD-specific antibody, S-specific antibody, and in-house ELISA, commercial sVNT, and in-house sVNT, against wild-type SARS-CoV-2. Three hundred and five samples were included in the analysis. The correlation between S-specific binding antibodies and in-house ELISA was 0.96 (95% CI 0.96–0.97) and between RBD-specific antibodies and in-house ELISA was 0.96 (95% CI 0.95–0.97). The Cohen’s kappa between in-house sVNT and the commercial test was 0.90 (95% CI 0.80, 1.00). If using 90% inhibition of sVNT as the reference standard, the optimal cut-off value of RBD-specific antibodies was 442.7 BAU/mL, the kappa, sensitivity, and specificity being 0.99, 99%, and 100%, respectively. The optimal cut-off value of S-specific antibodies was 1155.9 BAU/mL, the kappa, sensitivity, and specificity being 0.99, 100%, and 99%, respectively. This study demonstrated a very strong correlation between in-house ELISA and 2 commercial assays. There was also a very strong correlation between in-house and commercial SARS-CoV-2 sVNT, a finding of particular interest which will inform future research.


Results
Out of 153 participants who previously received 2 doses of CoronaVac® vaccine, 39 patients (25.5%) were male and the median age was 44 (IQR 30, 53) years. There were 305 specimens consisting of 77 samples at before the third dose of ChAdOx1 nCoV-19 vaccine, 76 samples at before the third dose of the BNT162b2 mRNA vaccine, 76 samples at 4 weeks after the third dose of ChAdOx1 nCoV-19 vaccine, and 76 samples at 4 weeks after the third dose of BNT162b2 mRNA vaccine. A total of 305 specimens were analysed.
The optimal cut-off value of binding specific IgG antibody using the commercial sVNT as a reference standard.
• At a cut-off level of 35% inhibition of commercial sVNT RBD-specific-IgG-test which designated 7.1 BAU/mL as a cut-off level of positive result, had 69% agreement with Cohen's kappa 0.04, 100% sensitivity, and 3% specificity. The S-specific-IgG-test, which designated 0.8 BAU/ mL as a cut-off level of a positive result, also had a 68% agreement with kappa 0, and 100% sensitivity.
After 4 sequential steps were applied, the area under the ROC (AUROC) curve of RBD-specific-IgG-test and S-specific-IgG-test was 0.99 and 0.97, respectively. ( Fig. 2A) The cut-off value for RBD-specific-IgG-test was 83.9 BAU/mL, which increased the agreement to 96%, with kappa 0.91. The sensitivity and the specificity were 96% and 96%, respectively. The optimal cut-off value for S-specific-IgG-test was 90.9 BAU/mL. By using this cut point, the agreement increased to 91% with kappa 0.82. The sensitivity and specificity were 94% and 89%, respectively. (Tables 1 and 2).
After 4 sequential steps were applied, the AUROC curve of RBD-specific-IgG-test and S-specific-IgG-test were 1 and 1, respectively. (Fig. 2B) RBD-specific-IgG-test had 99% agreement with kappa 0.99 at an optimal cut-off value of 442.7 BAU/ml with 99% sensitivity, and 100% specificity. At an optimal cut-off value of 1155.9 BAU/mL, S-specific-IgG-test had 100% agreement with kappa 0.99. The sensitivity of this assay was 100% and the specificity was 99%. The optimal cut-off value, % agreement, kappa, sensitivity and specificity for 80%, 85%, and 95% inhibition are shown in Tables 1 and 2.

Discussion
This study demonstrated a "very strong" correlation among SARS-CoV-2 IgG Quant II assay, Elecsys anti-SARS-CoV-2 S immunoassay, and S-specific antibodies by in-house ELISA, with a correlation above 0.95. Several studies have reported moderate to strong correlations between commercially available immunoassays [26][27][28] .
Although the correlation was very strong as regards direction, S-specific antibody by Elecsys anti-SARS-CoV-2 S immunoassay had a higher level than RBD-specific antibody detected by SARS-CoV-2 IgG Quant II assay at 4-weeks after, but not before the third dose vaccination. This finding has also been reported by Perkmann T. et al., which found that the correlation between Elecsys anti-SARS-CoV-2 S immunoassay and Abbott Architect RBD-specific antibodies was different at different time points after vaccination 29 . This emphasizes the importance of detailed interpretation of the results, particularly from different tests and at different time points. In addition, the antibodies from the in-house ELISA had lower levels than both S-specific antibodies by Elecsys Anti-SARS-CoV-2 S immunoassay and RBD-specific antibodies. (Fig. 1) A comparison between commercial and in-house techniques which detect antibodies to the spike protein but by different methods i.e. ECLIA in the former and ELISA in the latter, the antibody levels from in-house ELISA were lower. The correlation was weaker at the time point after the third dose than before. Similar findings were found when comparing the antibodies from in-house ELISA with the RBD-specific antibodies. In the case of the neutralizing antibodies, the correlation between in-house sVNT and commercial sVNT was also "very strong" with a correlation of 0.91, although there were www.nature.com/scientificreports/ some scattered results away from the regression line at 30-70% inhibition. There were satisfactory results of the in-house ELISA and also in-house sVNT with the commercially available tests. These tests may be cost effective alternative ways of detecting antibodies with acceptable performance. Several studies showed that vaccine efficacy and vaccine effectiveness correlated with neutralizing antibody levels for protection against symptomatic COVID-19 infection 9,24,30 . Protection against severe disease required lower neutralizing antibody levels than those conferring protection against mild disease 9 . Predictive models identifying a mean neutralization titer (relative to convalescent) of 1 correlate with 80% protective efficacy against SARS-CoV-2 infection, mostly due to SARS-CoV-2 variant B.1.177 24 . That study predicted vaccine efficacy against symptomatic infection of 50%, 60%, 70%, 80%, and 90% with the level of anti-spike IgG, anti-RBD IgG, normalized live-virus neutralization assay and pseudovirus neutralization assay respectively 24 . For example, 80% vaccine efficacy against symptomatic COVID-19 infection with mostly B.117 variants of SARS-CoV-2 was achieved with anti-spike IgG of 40,923 AU/mL, and 264 BAU/mL and anti-RBD-IgG of 63,383 AU/ml, and 506 BAU/mL, and NF50 for normalized live-virus neutralization assay of 247, and ID50 for pseudovirus neutralization assay of 57. No data to compare the vaccine efficacy and % inhibition of sVNT was published in that study. However, a correlation between sVNT and virus neutralization test (the plaque-reduction neutralization test; PRNT) has been reported 31 .  www.nature.com/scientificreports/ A cut-off level of positive result from SARS-CoV-2 IgG Quant assay and Elecsys anti-SARS-CoV-2 S immunoassay demonstrated no agreement with a cut-off level of seroconversion by commercial sVNT. After the 4 sequential steps were applied, the optimal cut-off values for SARS-CoV-2 IgG Quant assay and Elecsys anti-SARS-CoV-2 S immunoassay with 35% inhibition of commercial sVNT were 83.9 BAU/mL and 90.9 BAU/ mL, respectively. The level of 442.7 BAU/mL for SARS-CoV-2 IgG Quant assay and 1155.9 BAU/mL showed almost perfect agreement with 90% inhibition. We also performed tests at the cut points of 80%, 85%, and 95% inhibition, and the results are shown in Tables 1 and 2. The sensitivity and specificity were not reduced above 85% inhibition. However, the results must be interpreted with caution as the assays were tested against wild-type SARS-CoV-2, but not their variants.
The kappa statistic between in-house sVNT and the commercial sVNT was almost perfect. We have proposed an equation for switching between them. However, this equation may only be suitable for this study population and for research purpose when the costs are limited.
With straightforward technical requirements and lower operating costs, the determination of binding-specific antibody levels was more convenient for evaluating the immune response after vaccination against SARS-CoV-2 infection or immunity after infection. Furthermore, when long-acting antibodies (LAAB) are available for use for pre-exposure prevention of COVID-19 in certain populations, the antibody level may help to guide selection of eligible patients. For example, the Department of Disease Control, Ministry of Public Health of Thailand suggested prescribing LAAB for patients with end-stage renal disease (ESRD), kidney or other organ-transplant or bone marrow transplant who received immunosuppressive therapy, ESRD on hemodialysis or peritoneal dialysis, who had received at least 3 doses of vaccine against SARS-CoV-2 but had anti-spike IgG of/ or comparable to < 264 BAU/mL, as a first priority due to an imbalance between certain population and available LAAB 32 .
This study has several limitations. First, the binding specific antibodies and % inhibition from sVNT were tested against wild-type SARS-CoV-2 virus. The major variants of interest (VOIs), and variant of concerns (VOCs) required higher levels of antibodies than against wild-type SARS-CoV-2 30 . Interpretation and the cutoff levels from this study may not be applied the current situation where circulating virus are VOIs and VOCs. Second, the reference standard used in this study was the sVNT, which was not the gold standard for neutralizing antibody detection. Although there was a correlation between sVNT and the PRNT, the results must be interpreted with caution. Third, although in-house sVNT and in-house ELISA provide satisfactory results and has economic advantages when compared to the commercial assays, they require an overnight incubation of ACE-2 or the S antigen in ELISA plates. Therefore, they take more time to complete the assays compared to the ready to use commercial kits.
In conclusion, this study demonstrated a very strong correlation between in-house ELISA with 2 commercially assays available in Thailand, i.e. the SARS-CoV-2 IgG II Quant assay and the Elecsys anti-SARS-CoV-2 S immunoassay. However, when testing for binding specific antibody, the same tests must be used for longitudinal study as the antibody levels in each test were different (Fig. 1) There was also a very strong correlation between in-house sVNT and the commercial sVNT which is of great interest and will inform future research. These findings are very helpful for both research purposes, and to save costs with acceptable levels of outcome.

Methods
This cross-sectional study was conducted at Maharaj Nakorn Chiang Mai Hospital, a tertiary care hospital affiliated to Chiang Mai University from September to December 2021. For comparison of the tests, left-over serum samples taken from healthcare workers who enrolled onto a study into immunogenicity against SARS-CoV-2 after vaccination and had provided written informed consent for future studies were collected. In brief, participants were healthcare workers aged 18-60 years, previously received 2 doses of CoronaVac® vaccine, and received the third dose of vaccine either ChAdOx1 nCoV-19 (AZD1222) or BNT162b2 mRNA. Only serum from participants at before and 4 weeks after the third dose of vaccine were selected.

Laboratory assays
Binding antibody. The SARS-CoV-2 IgG II Quant assay (Abbott Laboratories Inc, IL, USA) 33 . This an automated, two-step immunoassay was designated for the qualitative and semi-quantitative detection of IgG antibodies to SARS-CoV-2 in human serum and plasma using chemiluminescent microparticle immunoassay (CMIA) technique. The amount of antibodies were presented as arbitrary units (AU)/mL and this assay measured the concentration of anti-RBD-WT (wild-type) IgG levels between 21 and 40,000 AU/mL. Those values were converted to binding antibody units (BAU)/ mL by multiplying by 0.142 per WHO recommendations 34 . The cut-off level for a positive result was ≥ 50 AU/ml (7.1 BAU/mL). 35 . Double-antigen sandwich is a major principle of the test. The reagent consists of antigens which predominantly captured anti-SARS-CoV-2 IgG, but also captured anti-SARS-CoV-2 IgA and IgM, levels being determined by electrochemiluminescence immunoassay (ECLIA). The values were presented as U/mL and this assay measured the concentration of anti-S-WT (wild-type) IgG levels between 0.4 and 250 U/mL. The samples with anti-SARS-CoV-2-S concentrations over the measurable range were diluted with Diluent Universal up to 1:100 dilution. Those values were converted to BAU/ mL by multiplying by 1.029 as per WHO recommendations (44). The cut-off level of a positive result was ≥ 0.8 U/mL (0.8232 ~ 0.8 BAU/mL) 34  www.nature.com/scientificreports/ Determination of specific S-binding IgG antibody by in-house ELISA. Antibodies specific to the spike protein were determined by indirect ELISA. Fifty microliters of 1 µg/mL spike proteins (Genscript, Piscataway, USA) in bicarbonate buffer (pH 9.6) were added to 96-well Maxisorp immunoplates (Thermo scientific, Roskilde, Denmark). After incubation overnight at 4 °C, plates were washed with washing buffer (0.05% Tween-20 (Calbiochem, Gibbstown, USA) in phosphate buffer saline (PBS) and blocked with 2% skimmed milk at 37 °C for 1 h. Fifty microliters of samples were diluted in the ratio 1:100 and serially diluted positive controls were added and incubated at 37 °C for 1 h. After washing, 50 µl of 1: 2000 goat anti-human IgG conjugated with horse radish peroxidase (HRP) (Invitrogen, Carlsbad, USA) were added and incubated at 37 °C for 1 h. After washing, 50 µl of tetramethylbenzidine (TMB) substrate (Life Technologies, Frederick, USA) were added and plates were incubated at room temperature for 30 min. The enzyme reaction was terminated using 0.2 M sulfuric acid and absorbance was read at 450 nm on a microplate reader (CLARIOstar®, Ortenberg, Germany). Antibody levels were determined from a standard curve, derived by serial dilution of WHO international standards for anti-SARS-CoV-2 immunoglobulin (NIBSC, UK) which was assigned an arbitrary unitage of 1000 BAU/mL. The theoretical curve for single antibody and antigen, and least-squares fit to logit-transformed data model was used to calculate the arbitrary units of samples as described previously 36 . The values were presented as BAU/mL and the cut-off level of seroconversion was set as ≥ 3 times of standard deviation of the negative test value which was 50.7 BAU/mL.

Neutralization assay. The SARS-CoV-2 NeutraLISA (EUROIMMUN Medizinische Labordiagnostika AG,
Lübeck, Germany) 37 . The concept of this test intended to mimic virus-host interaction utilizing recombinant RBD of the SARS-CoV-2 spike protein to detect antibodies that block the RBD binding to the hACE2 receptor by using an established ELISA method. The assay was performed following the manufacturer's instruction. The values identified in this study were presented as percent inhibition and the cut-off level of seroconversion was 35%.

The in-house SARS-CoV-2 surrogate virus neutralization test (in-house SARS-CoV-2 sVNT).
In-house SARS-CoV-2 sVNT was modified from the method previously described by Tan

Statistical analysis
Sample size calculation 38,39 . The sample size was calculated based on the comparisons between (1) S-specific-IgG-test and (2) commercial sVNT, and between (3) RBD-specific-IgG-test and commercial sVNT, which required the largest sample size in this study. Using the expected kappa of greater than 0.8 (e.g. 0.9), and probability of positive results for tests (1) and (3) were 0.75 and 0.75, and probability of seroconversion for test (2) was 0.7, respectively, with a two-sided alpha of 0.05, and a power of 80%, 312 samples were required.
Data analysis. Demographic data including age, gender, and underlying diseases were described as number (%), mean ± SD, and median (IQR) as appropriate. The correlation between 2 tests with continuous outcomes i.e.
To identify the value of binding specific IgG antibody which was correlated with 35%, 80%, 85%, 90%, and 95% inhibition of commercial sVNT, 4 sequential steps were performed. First, the regression of binary outcome on continuous outcome with logistic regression, (2) use of the receiver operating characteristic (ROC) curve to assess efficiency of classification, (3) identification of the cut-off point by Euclidian's index method for ROC curve, and (4) assessment of the accuracy of cut-off point for sensitivity and specificity using commercial sVNTs as a reference standard 42

Data availability
The datasets used and analysed during this study available from the corresponding author on reasonable request.